1. Field of the Invention
The present invention relates to software testing techniques and in particular to methods and associated systems and structures for automated software testing which integrates test definitions with source code of the software product under test.
2. Discussion of Related Art
Computer programs (also referred to herein as software products) evolve in an engineering process which is iterative. A software development process involving software design engineers is used to initially design and implement the software product. Testing of the initial design then verifies operation of the software product. Often the test and development processes repeat in an iterative manner to complete the development of the software product to a point where the program is substantially free of errors (also referred to as bugs).
In general, the program development process involves a programmer designing a solution for a problem described by a specification or functional requirement. The programmer proceeds first at a high-level envisioning abstract modules and functions to be performed. The high level abstractions are then further decomposed by the programmer (also referred to as software engineer) into corresponding lower levels in which the higher level design is implemented in a computer programming language. For example, programmers often utilize the C programming language, the C++ language, or the Java programming language for implementing a particular software product. Components of the software product written in such programming languages are often referred to as source code. The source code is created by programmers and stored in collections of files often referred to as source code files.
These programming languages are not directly executed by computers. Rather, computers execute instructions at a much lower level often referred to as machine code. The programs written in source code programming languages are then compiled, a step involving the translation of the source code files from the selected computer programming language to a machine code format suitable for execution by a general-purpose computer.
Programmers use a wide variety of sophisticated tools in performing software development tasks. Such software development tools often tightly integrate a variety of individual components utilized by software engineers in the development process. For example a text editor is often a first component utilized by software engineer to construct source code files for a particular product. In an integrated tool set, such a text editor may be particularly customized for the source programming language utilized by the programmer. In addition, higher level design tools and documentation generation tools may be integrated with the text editing process so as to document components of the eventual product design. Further tools include, compilers for translating source code into lower-level machine code operable on a particular computing environment. Still other tools include debugging tools used by the programmer to test the functionality of the designed software product while viewing the corresponding source code.
The program's source code therefore reveals it's structure and function in that the design of the source code represents internal structure of the software product. Such internal structure implements the requisite function of the product but is otherwise substantially transparent to the user of the software product. Rather, the user of the software product applies inputs to the software product in accordance with the specifications of the program and expects resultant outputs in accordance with the specifications.
Following initial development of a software product, an iterative process follows in which the software product is tested against specifications of its intended function. Problems found by such testing procedures are corrected by the programmer(s) by locating the reason for the problem using debugging tools and techniques, editing the source code files and re-compiling the source code. Following such debugging sessions, the program is re-tested verify proper operation or to locate other problems or new problems in need of "debugging." Such testing and debug cycles may be performed by an individual programmer or by teams often divided into subsets of program development engineers and testing engineers.
In general, testing of software involves applying data (generally referred to as stimuli) as input to the software product and determining that the program's behavior and output generated is in accordance with functional requirements and other specifications. When the testing engineer or testing team is separate and distinct from the program development engineer or development team, it is common that such testing is performed in a manner often referred to as "black box" testing. Such testing is referred to as black box in the sense that data or stimuli is applied as input and the output is evaluated without inspection, or even necessarily knowledge of, the internal structure of the software product reflected in the source code. In other words, the testing engineer views the software product as a black box, the internal structure and functions of which are not relevant to the test sequence and test result evaluation.
Conversely, testing procedures which utilize internal knowledge of the software product structure and function is often referred to as "white box" testing. The term white box testing refers to the fact that the tester has knowledge of the internal structure and design of the software product under test. For example, the testing engineer may direct particular types of stimuli or data as input to the software product knowing that the internal structure may be better exercised thereby. Or, for example, white box testing may involve design and application of a test sequence directed to exercising particular functions within the software product.
Tools for automating test procedures have evolved largely independent of the tools for programmers to design and debug software products. For example, it is known in the art to create and utilize automated testing tools which permit the test engineer to configure a sequence of input stimuli to be applied to the software product and to automate the gathering of resultant output data. Testing tools as presently known also provide the ability to automate the evaluation of the generated output to determine whether it complies with requirement specifications. Operation of computer programs are significantly dependent upon parameters outside of the specific input stimuli. Such external environmental conditions might, for example, include the state of the computer system on which the program runs. Testing tools have therefore included the ability to set up an external computing environment in preparation for execution over particular test. In other words environmental parameters of the computing system on which the program is run, as distinct from input stimuli applied to the software product program, may be configured by the automated test tool prior to performing a particular automated test sequence on the software product. For example, particular configuration parameters of the software product may be defined (e.g., in a configuration file) prior to performing a particular automated to sequence. Or, for example, configuration parameters associated with the computing system in which the software product operates may be configured prior to performing a particular automated to sequence.
In addition, present automated software test tools have included the ability to define test sequences is in a portable manner largely independent of the design methodology and tools used for program development. For example, present automated software test tools often use a script language to define input stimuli and environments and to operate programs to analyze the generated output. A test sequence and expected outputs defined in such a script language is often referred to as a test script. Such a script language is independent of the particular computer programming language used to implement the software product. The syntax and semantic of the script language is usually defined by the test tool itself and therefore parsed and otherwise processed by the test tool to perform desired test sequences defined in the test scripts. Present automated test tools are therefore portable in that the test sequences are portable across a wide variety of computing environments and are independent of the computer programming language used to implement the program.
In general, it is considered desirable for a software product to be tested in a manner which ensures significant coverage of the various execution paths within the software product. Most computer programs include significant sequences of logical controls (Boolean tests) which branch among various paths of the software product in response to results of tests within the program. One metric of the efficacy of software testing tools and techniques is the degree to which all possible branches in a program are tested by the test tool.
Automated test tools as presently known in the art are limited in their test capabilities, in large part, because of the limitations of "black box" testing. Though present test techniques provide significant automation for the test engineer in applying stimuli and gathering output results, they are limited in that through black box testing techniques is difficult to effectuate testing of the internal structures and functions of the software product. Such black box testing tools merely apply stimuli within a prescribed range of appropriate input values and verify the resultant output. These tools are therefore largely precluded from effectively testing internal data structures and internal functions which are not easily accessible through application of external stimuli. It is therefore difficult for black box testing techniques to assure test coverage which exercises all, or even a significant portion in some cases, of the possible branch paths through the software product. Rather, it is preferable to use "white box" test techniques to assure adequate test coverage of the internal structure and function of a program.
It is therefore problem in software testing to provide tools which automate software testing procedures in a manner which permits white box testing techniques to be applied to software product.
Is a further problem in the testing of software products to maintain white box test sequences in synchronization with changes made to the software product. Since software development tools and testing tools tend to be largely independent, is a problem for a software development engineering teams and software test teams to maintain synchronized versions of test sequences and corresponding software versions.
In view of the above discussion, it is evident that a need exists for improved software testing tools which permit both black box and white box testing techniques to be applied to software products in an automated fashion and in a manner which permits synchronization of the test sequences and changes made to corresponding software modules.